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ABSTRACT 
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level of mental processing is increased when students use prior 
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an interactive videodisc tutorial on biology. Data were analyzed 
using a pretest-posttest comparative group design. Results indicated 
that students in the learner-generated summaries group took more time 
to complete the task, yet scored lower on short-answe'" and 
multiple-choice tests than students in the multiple-ciioice and 
short-answer groups. It is concluded that replacing short-answer and 
multiple-choice questions in computer assisted instructional 
tutorials with generative processes such as learner-generated 
summaries does not facilitate learning outcomes. (88 references) 
(DB) 



********************************** 



* Reproductions supplied by EDRS are the best that can be made 

* from the origi lal document. 



U 8. DCPARTMf NT Of tOUCATION 
OHic© oi Educahonai ReMirch and lmprov«ment 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIQ 

□ This document has b««n reproduced a$ 
received from (he person or organizaiior 
Ortginatino i( 

D Minof changes have been made to improve 
reoroductiOn quality 

a Pomts ot view O' opinions slated m this docu- 
ment do not necessarily represent official 
OERI position or policy 



Leamer-Generated Summaries 

1 



Learner-Generated 
Summaries in Tutorial Courseware 



00 
i> 

o 

CO 

CO 

Bruce R. Harris 
Western Illinois University 
Grant V. Harrison 
Richard R Sudweeks 
Brigham Young University 



A paper presented at the Annual Meeting of ihe 
American Educational Research Association 
Chicago, Illinois, April 1991 

O Running Head: LEARNER-GENERATED SUMMARIES 

"PERMISSION TO REPRODUCE THIS 

\ MATERIAL HAS BEEN GRANTED BY 

S BEST COPY AVAILABLE HElF^^ 



o ■ 

ERIC 



2 TO THE EDUCATIONAL RESOURCES 

INFORMATION CENTER (ERIC).*' 



Learner-Generated Summaries 

2 

Learner-Generated 
Summaries in Tutorial Courseware 

Currently, the majority of commercial instructional sofu.arc for microcomputers, otherwise 
known as courseware, is in the tutorial and drill-and-practice modes (Cohen, cited in Jonassen, 
1988). According to Jonassen (1988), the typical instructional design of such courseware is 
similar to linear programmed instruction, where information is presented, followed immediately by 
recall or recogni'^ . questions about the information. TTiC learner's input is then immediately 
followed by feedback, which varies considera. \y in complexity and mode. 

Tutorial courseware allegedly enacts a dialog between the program designer and the learner. 
According to Soulier (1988), "dialog frames" present information to the learner, as well as carry 
out an interactive dialog/feedback between the learner and the computer. Soulier borrows the 
terminology of "dialog frames" from Alfred Pork (1980), who defined dialog as "a 'conversation' 
between a student and a teacher, where the student is at the computer display and the teacher is 
conducting the dialog through the medium of a computer program" (p. 15). 

Soulier (1988) explains that dialog frames in computer-based instruction (CBI) usually have 
three parts: rules, examples and interactions. The information or subject matter is usually 
presented in the first two parts. The third part is the practice, or as Soulier refers to it, the 
interaction function. 

According to Alessi and TioUip (1985), the most common method of CBI interaction is for the 
computer to pose questioiis the leamer must answer. Questions serve a number of purposes. 
They keep the student attentive to the lesson, force the student to practice, and assess the student's 
recall and comprehension. In addition, student responses to the questions determine the 
se-quencing for the renminder of the lesson. 

Questions can be categorized into two basic types: selected-response and constructed-response. 
Selected-response questions are those for which the student chooses the correct response or 
responses from a list of two or more optioiis. These include true/false, matching, multiple-choice, 
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and marking. Multiple-choice items are the most common type of selected-response questions 
used in CBI (Alessi & TroUip, 1985). Constructed-response questions are open-ended questions 
that require the student to produce rather than select a response. The three n^ajor types of 
constructed-response questions are completion, short-answer, and essay. In CBI, the most 
ftequent are completion and short-answer questions. 

In a chapter of a recent book on designs for microcomputer courseware, Jonassen (1988) states 
that one of the problems with most tutorial courseware is that the level of mental processing 
required of learners is too shallow. He claims this weakness is due to the multiple-choice and 
short-answer questions predomin ntly used in CBI interactions. He states that the nature of the 
interactivity is usually recognition and recall of information presented on the previous screen, with 
littie, if any, attempt to relate the information to prior knowledge, so that it will be meaningful and 
therefore retrievable by the learner. Jonassen (1988) states, 

Tutorial courseware is basically a mis-application of die programmed learning nxxiel of 
instructional design, which has been the dominate paradigm in the field of educational 
technology for nearly 3 decades. Since the piogrammed learning model is easily confused with 
the procedure or technique of programmed instruction, it is belter conceptually defined by the 
mathemagenic hypothesis. 

Matiiemagenic behaviors are "tiiose student activities that are relevant to the achievement of 
specified instructional objectives in specified situations or places," that is, tiiose which "give 
birth to learning" (Rothkopf, cited in Jonassen, 1988). These behaviors, according to the 
hypothesis, can be controlled or manipulated by specific design atuibutes of insUiiction, The 
form or structure of instmction or the activities stimulated by it induce die necessary cognitive 

operations to pnxiuce the desired learning The purpose of mathemagenic activities, such 

as inserted questions (the basis of programmed learning), is to control die way in which 
information is transformed and encoded into memory. It is therefore a reductive approach to 
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learning, which regards learners as active performers whose mental behavior should be strictly 
controlled by the activities imposed by the lesson, (p. 152) 

Jonassen continues by stating that one of the problems with computer applications of the 
mathemagenic nKxiel is that the level of processing normally produced by matiiemagenic behaviors 
(especially programmed learning) is too low or shallow. 

To solve the problem, Jonassen (1988) recommends tutorial courseware be developed based 
on a more constructive conception of learning, that is, instruction which focuses more on 
cognition, requiring deeper levels of mental processing. 

In Older to increase the level of mental processing, Jonassen (1988) recommends that tutorial 
courseware should enhance meaning of course information by stimulating die learners to call up 
and apply what they already know. He suggests that this deeper processing can be achieved by 
incorporating learning strategies into the courseware. These learning strategies sometimes referred 
to as cognitive strategies, are mental operations or procedures tiiat represent a wide range of 
cognitive activities including underlining main ideas, generating examples or analogies, 
summarizing, outlining, etc. which die si ident may use to acquire, retain, and retrieve different 
kinds of knowledge (Rigney, 1978). They are designed to increase die number of "links" between 
presented information and existing knowledge in order to enhance retention and to allow the 
student to process information generatively (Wittrock, 1974). They arc always performed by the 
leamer--at the initiation of die student or die instructional system. 

According to Weinstein and Mayer (1986), in recent years increased attention has been placed 
on the role of die learner as an active participant in die teaching-learning act. Many articles suggest 
that die effects of instruction depend partly on prior knowledge, and the learner's active cognitive 
processing during die learning process (Anderson, Spiro & Montague, 1977; Cook Mayer, 
1983; Dansereau, 1985; Jones, Amiran & Katims, 1985; Mayer, 1984; Ryan, 1981; Weinstein, 
1978; Weinstein & Underwood, 1985; Wittrock, 1974, 1978). Wittrock's (1974) generative 
learning hypothesis, assumes that when faced widi insuiictional stimuli, learners construct and 
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assign meaning to the stimuli on the basis of prior learning. The meaning gerjerated by learners for 
information they receive is individual and cannot be controlled by the author. According to 
Wittrock (cited in Jonasren, 1988), learning is not "a passive reception of someone else's 
organizations and abstractions" (p. 153). Instead, learning is an active, constructive process. 
According to Jonassen (1985), 

The purpose of generative strategies is to provide learners with active, constructive skills for 
proactively transferring prior knowledge. Meaning (knowledge, if you prefer) is learner- 
constructed, not media-controlled. These [learning] strategies, such as paraphrasing, 
generating questions, and imaging, are all individual processes for constructing meaning, 
(p. 31) 

How should an instructional designer integrate learning strategics into CBI? Rigney (1978) 
describes a framework of several approaches for integrating learning strategies into an instructional 
system. Some are detached from the content, such as explicit instructions to generate a question or 
form an image about the material, beaming strategies may also be embedded in the content, so that 
the learners must perform the mental operations in order to acquire the material, such as a question 
that requires the learners to relate new material to prior learning or solve a problem applying a 
concept before the next information will be given. Rigney draws an additional distinction-crucial 
to computer-based instruction-between instructional system assigned strategies (ISA), which 
recommend when the student should use one or more of the strategies as a part of the courseware, 
and student-assigned (SA) strategies, which are initiated by the students without guidance from the 
system (Allen & Merrill, 1985). The relationship between these two dimensions is shown as a 
matrix in Figure 1. 

Each cell in the matrix represents a different approach for integrating learning strategies. Cell 
"D" represents instruction which incorporctej, the cognitive learning strategy that the designer 
considers the best for each particular phase of learning into the design of the learning sequence. 
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Figure 1. Alternative Approaches for Integrating Cognitive Leaming 
Strategies (Rigney, 1978). 

Cell "A" represents instruction in which the designer makes no attempt to induce any particular 

cognitive leaming strategy, leaving the selection and the control with the individual learner. Cell 

"C" represents the condition which encourages the learner to choose from a number of possible 

strategies that are embedded within the instructional system. Cell "B" represents the situation in 

which the instructional system suggests that the learner use a leaming strategy that is detached firom 

the instruction (Eucker, 1984). 

Jonassen (1985) explains that microcomputers are especially amenable to the inclusion of 
leaming strategics because they can accept, store, and manipulate a variety of input, and they can 
insist on a response before allowing the learner to proceed, something which traditional print media 
cannot do. For instance, after presenting the infonnation, the computer system could direct the 
learner to go through a particular leaming strategy, with the computer functioning as an "electronic 
notebook." The learner can key in a response which the computer evaluates for quantity or 
existence of key concepts and sometimes manipulates through rearranging or mapping and then 
stores for use by learners as a review or retrieval strategy. Jonassen (1988) believes that assessing 
user input will ensure higher levels of processing than the multi-option recognition or recall tasks 
included in most tutorial courseware. 

Jonassen (1988) recommends that "the simplest method for integrating [leaming] strategies in 
courseware is to replace the adjunct, mathemagenic activities that are normally included as practice 
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in courecware with specific information processing or perhaps inetalearning strategies" (p. 160). 

More specifically he suggests: 

For example, rather than inserting multiple-choice questions to test immediate recall or 
comprehension of information in a program, you might periodically insert any of the following 
directions: Summarize in your own words the ideas presented; recall and record key ideas and 
use them to create analogies, outlines, or cognitive maps; draw a picture or generate a mental 
image of the subject matter, or list the implications of the material that you are studying 
(p. 160). 

Jonassen's approach is an example of cell "D" (Instructional system-assigned/ embedded 
approach) of Rigney's matrix in Figure 1. 

In discussing the rationale for replacing multiple-choice and short-answer questions in CBI 
interaction frames, Jonassen (1988) explains that "only deeper, semantic processing of information 
requires the learner to access prior knowledge in order to interpret new material" (p. 153). He 
makes the point that integrating cognitive strategies into CBI interactions will facilitate higher level 
learning outcomes, such as comprehension and understanding, and will also increase the amount 
of information that is recalled. Because learners assign more meaning to the information presented 
during the tutorial, they will be able to remember more. Jonassen continues. 

It is exactly this level of meaningful learning that is most frequently missing from mtorial types 
of courseware. The emphasis is on practice of associations in working memory based only on 
information recently presented in the courseware. In interacting with the tutorial courseware, 
learners are too seldom required to access prior knowledge in order to interpret the information 
that is presented, (p. 153) 

The primary purpose of this study was to e- imined the effects of replacing multiple-choice and 
short-answer questions in tutorial courseware practice interactions with a learning strategy (learn er- 
generated summaries) on three types of verbal information learning outcomes. The study was 
designed to investigate empirically the approach recommended by Jonassen (1988) to integrate 



8 



Leamer-Generated Summaries 

8 

learning strategies by replacing the traditional short-answer and multiple-choice questions in CBI. 
interactions. 

A verbal information learning outcome can be defined as the kind of knowledge the learner is 
able to state from menrrary. As Gagne, Briggs and Wager (1988) explain, "All of us have a great 
deal of verbal information or verbal knowledge. We have readily available in our memories many 
commonly used items of information such as the names of months, days of the week, letters, 
numerals, towns, cities, states, countries, and so on" (p. 47). Verbal information may be recalle-d 
or stated, but not necessarily understood. As Tessmer, Jonassen & Caveriy (1989) explain. 
The learner acquires knowledge of a concept, mle, principle, or procedure, but does not 
necessarily understand the concept, rule or principle. Nor can we be assured that the learner 
can use the rule or concept even though they can say it. Verbal information refers to the 
statement of facts, rules, concepts, etc. The statement of information is different from the use 
or application of the information. . . . If the learner only memorizes the information so that 
she or he may later state it, the outcome is verbal information (pp. 33-34). 
There are many specific types of learning outcomes within tlie domain of verbal knowledge or 
verbal information. For example, the ability to describe the procedure for making withdrawals 
from a bank, recall the quadratic formula, state the definition of hegemony, or recognize Robert's 
Rule of Order regarding a quorum (Tessmer, Jonassen & Caveriy, 1989). Another verbal 
information learning outcome is the capacity to explain or describe the gist of paragraph-long 
passages of information (Gagne, Briggs & Wager, 1988). 

Even though verbal information learning outcomes are tiie lowest category in most learning 
taxonomies (see Bloom, Engiehart, Furst, Hill, & Krathwohl, 1956 and Gagne, 1985), the 
usefulness of a student's "information store" should not be minimized (Nitko, 1983). As Nitico 
explains, "A base of knowledge in a subject is important (a) when learning complex intellectual 
skills, (b) in communicating with others in tne context of everyday living, and (c) as a 'medium' or 
'carrier" of thought for more reflective and creative tiiinking and problem solving" (p. 21 5). 
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The present study assessed the effects of integrating the learning strategy of learner-generated 
summaries in tutorial courseware on three types of learning outcomes within the domain of verbal 
information: short-answer recall, recognition, and recall of organized information. 

Although several studies within the last few years have investigated the effects of integrating 
individual learning straiegies in instructional systems and developing instructional systems that 
train learners how to use a particular learning strategy (Weinstein & Mayer, 1985; Dansereau et al ., 
1979; O'Neil, 1978; O'Neil & Spielberger, 1979), the practice of integrating learning strategies in 
CBI interactions has not received much attention in the literature*,. The few studies which have been 
conducted had a different focus than the present study. For example, these studies either dealt witii 
a different learning outcome than verbal information or they tried to validate a different type of 
learning strategy (for example, see Eucker, 1984; Allen, 1982; Wilshire, 1990). The results from 
each of these studies did not support tne hypotiiesis that integrating a learning sti-ategy in CBI 
would be nK)re effective in facilitating learning than the traditional approach used by the 
comparison group(s). 

Ill a recent article Mayer (1989) concluded that having learners use the cognitive strategy of 
building conceptual mental models (used to promote more deeper, meaningful, systematic mental 
processing) actua'V reduced verbatim retention of verbal information. His article examined the 
effects of using conceptual models on three learning outcomes: recall of conceptual information, 
verbatim retention, and generation of creative solutions on transfer problems. In a review of 20 
studies involving 31 tests, results consistently indicated tb t this strategy improved recall of 
conceptual information, decreased verbatim retention, and increased creative solutions to transfer 
problems. Mayei-'s article and some of the studies cited previously provide some evidence which 
does not support the premise stated by Jonassen (1988) that integrating a learning strategy in CBI 
by replacing short-answer and multiple-choice questions will facilitate more recall of verbal 
infonnation. The present study is innovative in that it investigated the specific effects of integrating 
leamer-generated summaries on verbal infonnation learning outcomes. 
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Several studies have examined different aspects of student-generated summaries (Annis, 1985; 
Brown, Campione, & Day, 1981; Day, cited in Brown, Campione, & Day, 1981; Ballesteros, 
1986); however, none have examined the specific effects of integraiing this strategy in CBI 
interactions on verbal information learning outcomes. 

The following hypotheses were addressed in this study: 
Hypotheses 

1 . Learner-generated summaries in tutorial courseware will be more effective in facilitating 
learners' ability to recall and recognize isolated facts or parts of information than multiple- 
choice or short-answer questions. 

2. Learner-generated summaries in tutorial courseware will be more effective in facilitating 
learners' ability to recall organized information than multiple-choice or short-answer questions. 

Method 

Subjects 

Subjects were obtained from an undergraduate computer course (Computers in Education) 
offered in the College of Education at a university in Utah. They were required to complete the 
intervention for this study as one of their assignments in the course. Sixty-nino students 
participated in the study: 57 females and 12 males. The subjects were undergraduates (expect for 
six graduate students) enrolled in teacher education majois. 
Materials 

The intervention for all treatment groups consisted of an interactive videodisc tutorial on 
developmental biology, developed by WICAT Incorporated in 1979, under a grant from the 
National Science Foundation (Bundcrson, Olsen & Baillio, 1981). The hardware consisted of a 
Macintosh-based interactive videodisc system. 

The instruction, which was developed for university-level students, dealt primaiily with verbal 
information learning outcomes. This explanative material (Mayer, 1989) consisted of an 
introduction to basic developmental biology concepts such as DNA, RNA, genes, chromosomes, 
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etc. It was selected for several reasons. First, the researcher believed that most students had very 
little prior knowledge about the content, which was desirable determining whether a learning effect 
occurred as a result of the treatment. Second, the instruction on the videodisc had been effectively 
developed using the traditional tutorial courseware approach: an instructional segment followed by 
short-answer and multiple-choice questions as the form of practice. Finally, since the content area 
was not related to the course in which students were enrolled, it was expected that they would not 
make extensive use of self-initiated learning strategies during the intervention as they might if the 
information held strong interest or high motivation for them. The addition of self-initiated learning 
strategies could have seriously confounded the results of the study. 

The interaction frames and the courseware that controlled the videodisc were developed on the 
computer by use of the authoring program The Best of Course of Action, developed by 
Authorware, Incorporated (Authorware, 1987). 
Instruments 

A pretest was developed using 20 itenis selected from a test developed by WICAT Inc. and a 
professor of Biology at Brigham Young Univeisity, based on the information presented in the 
videodisc. Multiple-choice, matching, true/false, and short-answer test items were selected to 
assess students' preexisting knowledge of developmental biology in terms of verbal information 
learning outcomes. 

Three posttests were developed measuring the three learning outcomes: (a) a short-answer 
recall test, (b) a multiple-choice and matching test assessing recognition, and (c) a restricted- 
response essay test assessing recall of organized information. The material presented on the 
videodisc was analyzed to identify all the knowledge elements presented during the instruction and 
test items were written for each. The items for the short-answer and multiple-choice tests assessed 
exacUy the san>e knowledge elen^ients; however, to prevent cueing, the items on the multiple-choice 
test were randomly reordered. The short-answer recall and recognition posttests were developed to 
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measurc the degree to which learners were able to remember all the information presented during 
the instruction. 

The four questions on the essay test were written ith the intent to assess how much organized 
information the learners could recall. One of the purposes of including an essay test was to 
determine if students who wrote summaries would perform better on an explanation test than 
students who selected the correct answer or constructed a short answer. 

Internal consist ^ncy reliability measures were computed on the test scores to assess the 
reliability of the results. The reliability (Cronbach Alpha) coefficients for the pretest, short-answer 
and multiple-choice tests were .72, .89, and .80, respectively. 

Three laters judged the students' answers to the restricted-response essay questions. Table 1 
shows the results of interrater reliability coefficients computed for each essay question and the total 
score for tnc essay test. 

As an estimate of content-related evidence of validity, three raters (two graduate students and 
one undergraduate student) were asked to review the instruction and determine to what degree the 
tests actually assessed the material presented in the instruction. Two raters were very familiar with 
the material presented in the instruction. The raters were asked by the researcher to make a 
quantitative judgment concerning the evidence of content-related validity for the results of the 
instt-uments using a 10-point scale, 10 being the highest and 0 being the lowest. As the raters' 
judgments were eight, eight, and nine, their judgments were highly consistent, and the validity 
rating for the results of the instruments was very high. 
Procedure 

Administration of intervention. To reduce the internal validity threat of experimenter bias, two 
male proctors who did not know the specific nature of the study or the hypotheses administered the 
study. They administered the intervention to each student individually, during a prearranged one- 
hour time slot When the students arrived, a proctor briefly explained the general nature of tiie 
study and then requested tiie students to sign a participation consent form and complete the pretest 

o 13 
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Table 1 

Estimated Interrater Reliability Coefficients for Each Essay Question and the Total Score for the 
Essay Test 



Essay Alpha 

Question Coefficient 

1 .969 

2 .978 

3 .978 

4 .982 
Total Score .987 
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After the students completed the pretest, the proctor provided some basic explanations on how to 
use the instructional system, then selected the appropriate treatment group for v hich each student 
had been assigned. 

Following a few introductory frames in the computer piogram, the students entered their names 
and social security numbers. The program then presented four segments of instruction. After the 
fu-st segment, which consisted of several still frames and a r »tion sequence, students assigned to 
Group 1 answered multiple-choice questions, smdents in Group 2 wrote short-answer questions 
v/hile students in Group 3 composed a summary about the infomiation presented, t: iie students in 
Group 1 or 2 selexjted tlie correct answer, the system provided a short statement affiraiing the 
correct selection or answer. However, if the wrong answer was selected or constructed, the 
students were informed the answer was incorrect and then were given the correct answer. If the 
students were in Group 3, the system presented a model summary as a forai of feedback 
immediately after tlie students typed in their summary. Students were allowed unlimited time to 
answer questions, but were not allowed to search back lO the text for answers. 

This process was the same for the remaining three segments of instruction. Students in Group 
1 answered a total of 35 multiple-choice questions; students in Group 2 answered a total of 38 
short-answer questions; students in Group 3 wrote four summaries during tt»c intervention. The 
computer program recorded the time students started the intervention and the time they completed 
the instruction. In addition, the program recorded the number of practice items which students in 
Groups 1 and 2 scored correcdy and the self-generated summaries students in Group 3 typed. 

After completing the intervention, students completed the recall test, which preceded the 
recognition test to avoid prompting or cueing. Following the recall test, students completed the 
recognition tesc and then the restricted-response essay test, after which they were thanked for 
participatirig in the study. It took an average of about 60 minutes for students to complete the 
entire inte»vcntion. 
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Training of proctors. The proctors (a graduate student and an undergraduate student) were 
trained by the researcher and provided with a sample dialogue to ensure consistency in 
administering the intervention. 

Procedures for grading essay tests. The primary researcher and the two proctors rated tlie 
essay tests. To increase the interrater reliability of the results of the test, the two pioctors received 
a two-hour training session conducted by the primary researcher. 

The ra*'*'^ followed the procedures for rating the essay tests recommended by Mehrens and 
Lehmann (1984) and Gronlund and Linn (1990): (a) the tests were graded one question at a time 
for all tests, (b) the tests were randomly shuffled after grading each question, (c) the tests for all 
students were graded without interruption, (d) the tests were graded anonymously, (e) the rating 
criteria were applied as consistently as possible, and (0 the mechanics of expression were judged 
separately from the content 

Scoring procedures and data entry. To ensure that the pretest, recall and recognition tests were 
scored correctly, each proctor graded all the tests using a separate answer sheet for each student's 
test. Each proctor then entered the data from the answer sheets into the computer. The researcher 
compared the sets of data, using a comparison program on the VAX mainframe computer, and 
corrected discrepancies. The verified, corrected data were used in the fxnal data analyses. 
Research Design 

The research design was a Pretest-Posttest Comparison Group Design. The students were 
randomly assigned to one of tliree treatment groups. The instruction delivered during the 
intervention was the same for all three treatment groups-four segments consisting of video 
sequences and several still frames on developmental biology. 

The independent variable was the type of learner response required during the CBI interaction 
following the instructional frames in the tutorial courseware. Several dependent variables were 
included: short-answer recall measured by the short-answer test, recognition measured by the 
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multiple-choice test, and recall of organized information measured by the restricted-response essay 
test. 

Statistical Procedures 

The hypotheses were addressed using the following procedures. Because there was more than 
one dependent variable, a multivaiiate analysis of variance was used to determine if there was a 
statistically significant difference between treatment groups with respect to at least one dependent 
variable. The multivariate analysis was followed by separate univariate one-way analyses of 
variance for each dependent variable. If an analysis of variance resulted in a significant difference 
between treatment groups, the Newman-Keuls multiple comparison method was used to determine 
which means differed firom the other. One-way analysis of variance was used to analyze die 
pretest scores. 

Results 

Of the 69 subjects, 23 were in Group 1 (multiple-choice questions), 22 were in Group 2 
(short-answer questions), and 24 were in Group 3 (learner-generated summaries). Means, 
standard deviations and ranges were computed for the number of correct answers on the pretest (20 
possible) and for each of the dependent variables: (a) number of correct answers on the recall test 
(35 possible), (b) number of correct answers on the recognition test (25 possible), and (c) number 
of correct answers on the essay test (32 possible). Table 2 summarizes the results of these 
computations. 

One-way analysis of variance showed that the differences between means on the pretest scores 
were not statistically significant, an indication that the three groups were equivalent in terms of 
preexisting knowledge about the material presented in the instruction , F (2, 66) = .04, p = .96. 

The multivariate analysis of vaiianc^ showed a significant difference between the treatment 
groups with at least one dependent variable using Wilk's Lamba (Exact F = 3.37, df = 6, 128, p = 
.005). 
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Table 2 

Group Means, Standard Deviations, and Ranges for Pretest Scores, Recall Test, Recognition Test, 
and Essay Test 



Treatment 


Pretest 


Recall 


Recognition 




Group 


Scores 


Test 


Test 


Test 


iVlUlupiC-v^llUlvC 










Mean 


9.0 


25.8 


21.5 


14.3 


SD 


3.3 


5.7 


3.3 


5.3 


Range 


4-16 


12-33 


10-25 


5.0-25.3 


Short-Answer 










Mean 


9.4 


27.6 


22.0 


13.9 


SD 


4.0 


6.4 


2.1 


4.8 


Range 


2-17 


11-35 


18-25 


4.3-21.7 


Learner-Generated 










Mean 


9.1 


20.8 


18.5 


13.2 


SD 


4.0 


7.3 


4.3 


5.6 


Range 


3-18 


12-35 


9-25 


2.3-30.3 


Note: The range of scores for the essay test is reported in decimals because the scores given by 


three raters were averagexl to create a 


single rating for each examinee. 
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The results of the univariate analysis of variance showed a significant difference between the 
means of the three treatment groups on the results of the short-answer test, F (2,66) = 6,77, p - 
.0021. Results of a Newman-Keuls multiple comparison (p < .05) indicated that the mean for the 
leamer-generated summaries group was significantly lower than the mean of the multiple-choice 
group and the mean of the short-answer group. There was no significant difference between the 
meaiis of the short-answer and multiple-choice groups. 

An analysis of variance on the results of the recognition test also showed a significant 
difference between the three treatment groups, F (2,66) = 7.31, p = .0014. As with the short- 
answer test, a Newman-Keuls multiple comparison (p < .05) indicated the mean of the leamer- 
generated summaries group was. significandy lower than the mean of the multiple-choice group and 
the mean of the short-answer group. Like the previous analysis, there was no significant 
difference between the means of the short-answer and multiple-choice groups. These results do 
not support and are opposite to those predicted in Hypothesis One, Nvhich states the use of leamer- 
generated summaries in tutorial courseware will have a greater faciliiiative effect on learners' ability 
to recall and recognize isolated facts or parts of information than multiple-choice or short-answer 
questions. 

An analysis of variance showed that the differences between the treatment means on the essay 
test scores were not statistically significant, F (2,66) = .25, p = .780. These results do not support 
I lypothiesis Two, which states that the use of leamer-generated summaries in tutorial courseware 
will more effectively facilitate learners' ability to recall organized infomiation than multiple-choice 
or short-answer questions. 

Discussion 

The results of this study did not support the first hypothesis. Students in the leamer-generated 
summaries group took significantly more time to complete the intervention and yet scored 
significantly lower on the short-answer and multiple-choice tests than students in the multiple- 
choice and short-answer questions groups. 
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It was also hypothesized that students in the learner-generated summaries gioup would score 
higher on the essay tests than students in the other groups, since part of the task of generating a 
summary involves recalling organized information. However, the results of this study showed that 
students in the multiple-choice and short-answer groups performed as well as students in the 
learner-generated summary group. 

What are the theoretical implications of these findings? First, for the particular verbal 
information learning outcomes assessed in this study, students may need the practice items of 
short-answer and multiple-choice questions to provide the necessary repetition required for 
immediate recall and recognition (Gagne, 1985; Gagne, Briggs & Wager, 1988). Second, the 
short-answer and multiple-choice response items may have highlighted the key knowledge 
elements presented in the instruction which focused the learners' attention on the information that 
was really important. Third, the short-answer and multiple-choice practice items provided 
feedback on each of the major knowledge elements presented in the instmction so learners knew if 
they missed some particularly important information. The students in the learner-generated 
summaries group did not receive specific feedback on each of the major knowledge elements 
presented in the instruction, which may have had an influence. Fourth, another theoretical issue 
which may have had some influence on the results of this study is die degree of instructional 
alignment between the learning strategy and the learning outcomes (Cohen, 1987). Perhaps using 
learner-generated summaries was not die most appropriate strategy to use for the particular learning 
outcomes measured. Leanicr-generated summaries may have facilitated other higher-level learning 
outcomes that were not measured in this study, such as understanding, transfer, synthesis, etc. 
Finally, another possible explanation which may have influencexl the results of this study was the 
degree to which students wrote effective summaries. Perhaps the subjects in this study were much 
more familiar with responding to short-answer and multiple-choice items and hadn't received much 
practice in writing summaries of explanatory material. 
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What are the major contributions of this study? The results of this study show that replacing 
short-answer and multiple-choice questions in tutorial courseware with learner-generated 
summaries docs not facilitate the specific learning outcomes of short-answer recall, recognition, 
and recall of organized information for the material used in this study. In fact, the results of this 
study sho'.v that using Reamer-generated summaries in place of short-answer and multiple-choice 
questions is not only less effective, but also less efficient for the particular material presented in 
this study. Students who generated their own summaries not only scored significantly lower than 
students who answered short-answer and multiple-choice questions, but they also took 
significandy longer to complete the instruction. 
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